An Electroencephalographic Investigation of the Encoding of Sound Source Elevation in the Human Cortex

Sound localization is of great ecological importance because it provides spa- tial perception outside the visual field. However, unlike other sensory systems, the auditory system does not represent the location of a stimulus on the level of the sensory epithelium in the cochlea. Instead, the position of a sound source has to be computed based on different localization cues. Different cues are informative of a sound sources azimuth and elevation, which, when taken together, describe the sources location in a polar coordinate system. There is a body of knowledge regarding the acoustical cues and the neural circuits in the brainstem required to perceive sound source azimuth and elevation. However, our understanding of the encoding of sound source location on the level of the cortex is lacking especially what concerns elevation. Within the scope of this thesis, we established an experimental setup to study auditory spatial perception while recording the listeners brain activity using electroencephalography. We conducted two experiments on the encoding of sound source elevation in the human cortex. Both experiments results are compatible with the hypothesis that the cortex represents sound source elevation in a population rate code where the response amplitude decreases linearly with increasing elevation. Decoding of the recorded brain activity revealed that a distinct neural representation of differently elevated sound sources was predictive of behavioral performance. An exploratory analysis indicated an increase in the amplitude of oscillations in visual areas when the subject localized sounds during eccentric eye positions. More research in this direction could help shed light on the interactions between the visual and auditory systems regarding spatial perception. The experiments presented in this dissertation are, to our knowledge, the first studies that demonstrate the encoding of sound source elevation in the human cortex by using a direct measure of neural activity (i.e., electroencephalography).:Abstract . . . . . . . . . . . . . . . . . . . . . . 1 Zusammenfassung . . . . . . . . . . . . . . . . . . . . . . 7 1 Electroencephalography 13 1.1 Event Related Potentials and Oscillations . . . . . . . . . . . . 13 1.2 Comparison to other Methods . . . . . . . . . . . . . . . . . . . 14 1.3 EEG Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4.2 Referencing . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.4.3 Eye Blinks . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.4 Epoch Rejection . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5.1 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.5.2 Nonparametric Permutation Testing . . . . . . . . . . . 26 1.5.3 Source Separation . . . . . . . . . . . . . . . . . . . . . . 28 2 Sound Localization in the Brain . . . . . . . . . . . . . . . . . . . . 31 2.1 The Spatial Perception of Sound . . . . . . . . . . . . . . . . . . 32 2.1.1 Interaural Cues . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.2 Spectral Cues . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 Brain Mechanisms for Sound Localization . . . . . . . . . . . . 37 2.2.1 Auditory Pathway . . . . . . . . . . . . . . . . . . . . . 38 2.2.2 Extracting Localization Cues . . . . . . . . . . . . . . . 40 2.2.3 Neural Representation of Auditory Space . . . . . . . . 42 2.2.4 The Dual Pathway Model . . . . . . . . . . . . . . . . . 45 2.2.5 A Dominant Hemisphere for Sound Localization? . . . 47 2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3 A Free Field Setup for Psychoacoustics 51 3.1 Design of the Experimental Setup . . . . . . . . . . . . . . . . . 51 3.1.1 Loudspeakers . . . . . . . . . . . . . . . . . . . . . . . . 54 3.1.2 Processors . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.3 Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.4 Coordinate Systems . . . . . . . . . . . . . . . . . . . . 56 3.2 Operating the Setup . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.2.1 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.2.2 Loudspeaker Equalization . . . . . . . . . . . . . . . . . 59 3.3 Head Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . 61 3.3.1 Landmark Detection . . . . . . . . . . . . . . . . . . . . 62 3.3.2 Perspective-n-Point Problem . . . . . . . . . . . . . . . 62 3.3.3 Camera-to-World Conversion . . . . . . . . . . . . . . . 63 3.4 A Toolbox for Psychoacoustics . . . . . . . . . . . . . . . . . . 64 4 A Linear Population Rate Code for Elevation 67 4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 69 4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 70 4.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 70 4.2.3 Elevation Encoding . . . . . . . . . . . . . . . . . . . . . 72 4.2.4 Effect of Eye-Position . . . . . . . . . . . . . . . . . . . . 74 4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5 Decoding of Brain Responses Predicts Localization Accuracy . . . 81 5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 82 5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 83 5.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 84 5.2.3 Decoding Brain Activity . . . . . . . . . . . . . . . . . . 86 5.2.4 Topography of Elevation Encoding . . . . . . . . . . . . 88 5.2.5 Elevation Tuning . . . . . . . . . . . . . . . . . . . . . . 89 5.2.6 Hemispheric Lateralization . . . . . . . . . . . . . . . . 91 5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 A Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 B Publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

An Electroencephalographic Investigation of the Encoding of Sound Source Elevation in the Human Cortex

Sound localization is of great ecological importance because it provides spa- tial perception outside the visual field. However, unlike other sensory systems, the auditory system does not represent the location of a stimulus on the level of the sensory epithelium in the cochlea. Instead, the position of a sound source has to be computed based on different localization cues. Different cues are informative of a sound sources azimuth and elevation, which, when taken together, describe the sources location in a polar coordinate system. There is a body of knowledge regarding the acoustical cues and the neural circuits in the brainstem required to perceive sound source azimuth and elevation. However, our understanding of the encoding of sound source location on the level of the cortex is lacking especially what concerns elevation. Within the scope of this thesis, we established an experimental setup to study auditory spatial perception while recording the listeners brain activity using electroencephalography. We conducted two experiments on the encoding of sound source elevation in the human cortex. Both experiments results are compatible with the hypothesis that the cortex represents sound source elevation in a population rate code where the response amplitude decreases linearly with increasing elevation. Decoding of the recorded brain activity revealed that a distinct neural representation of differently elevated sound sources was predictive of behavioral performance. An exploratory analysis indicated an increase in the amplitude of oscillations in visual areas when the subject localized sounds during eccentric eye positions. More research in this direction could help shed light on the interactions between the visual and auditory systems regarding spatial perception. The experiments presented in this dissertation are, to our knowledge, the first studies that demonstrate the encoding of sound source elevation in the human cortex by using a direct measure of neural activity (i.e., electroencephalography).:Abstract . . . . . . . . . . . . . . . . . . . . . . 1 Zusammenfassung . . . . . . . . . . . . . . . . . . . . . . 7 1 Electroencephalography 13 1.1 Event Related Potentials and Oscillations . . . . . . . . . . . . 13 1.2 Comparison to other Methods . . . . . . . . . . . . . . . . . . . 14 1.3 EEG Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4.2 Referencing . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.4.3 Eye Blinks . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.4 Epoch Rejection . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5.1 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.5.2 Nonparametric Permutation Testing . . . . . . . . . . . 26 1.5.3 Source Separation . . . . . . . . . . . . . . . . . . . . . . 28 2 Sound Localization in the Brain . . . . . . . . . . . . . . . . . . . . 31 2.1 The Spatial Perception of Sound . . . . . . . . . . . . . . . . . . 32 2.1.1 Interaural Cues . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.2 Spectral Cues . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 Brain Mechanisms for Sound Localization . . . . . . . . . . . . 37 2.2.1 Auditory Pathway . . . . . . . . . . . . . . . . . . . . . 38 2.2.2 Extracting Localization Cues . . . . . . . . . . . . . . . 40 2.2.3 Neural Representation of Auditory Space . . . . . . . . 42 2.2.4 The Dual Pathway Model . . . . . . . . . . . . . . . . . 45 2.2.5 A Dominant Hemisphere for Sound Localization? . . . 47 2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3 A Free Field Setup for Psychoacoustics 51 3.1 Design of the Experimental Setup . . . . . . . . . . . . . . . . . 51 3.1.1 Loudspeakers . . . . . . . . . . . . . . . . . . . . . . . . 54 3.1.2 Processors . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.3 Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.4 Coordinate Systems . . . . . . . . . . . . . . . . . . . . 56 3.2 Operating the Setup . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.2.1 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.2.2 Loudspeaker Equalization . . . . . . . . . . . . . . . . . 59 3.3 Head Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . 61 3.3.1 Landmark Detection . . . . . . . . . . . . . . . . . . . . 62 3.3.2 Perspective-n-Point Problem . . . . . . . . . . . . . . . 62 3.3.3 Camera-to-World Conversion . . . . . . . . . . . . . . . 63 3.4 A Toolbox for Psychoacoustics . . . . . . . . . . . . . . . . . . 64 4 A Linear Population Rate Code for Elevation 67 4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 69 4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 70 4.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 70 4.2.3 Elevation Encoding . . . . . . . . . . . . . . . . . . . . . 72 4.2.4 Effect of Eye-Position . . . . . . . . . . . . . . . . . . . . 74 4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5 Decoding of Brain Responses Predicts Localization Accuracy . . . 81 5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 82 5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 83 5.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 84 5.2.3 Decoding Brain Activity . . . . . . . . . . . . . . . . . . 86 5.2.4 Topography of Elevation Encoding . . . . . . . . . . . . 88 5.2.5 Elevation Tuning . . . . . . . . . . . . . . . . . . . . . . 89 5.2.6 Hemispheric Lateralization . . . . . . . . . . . . . . . . 91 5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 A Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 B Publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

An Electroencephalographic Investigation of the Encoding of Sound Source Elevation in the Human Cortex

Sound localization is of great ecological importance because it provides spa- tial perception outside the visual field. However, unlike other sensory systems, the auditory system does not represent the location of a stimulus on the level of the sensory epithelium in the cochlea. Instead, the position of a sound source has to be computed based on different localization cues. Different cues are informative of a sound sources azimuth and elevation, which, when taken together, describe the sources location in a polar coordinate system. There is a body of knowledge regarding the acoustical cues and the neural circuits in the brainstem required to perceive sound source azimuth and elevation. However, our understanding of the encoding of sound source location on the level of the cortex is lacking especially what concerns elevation. Within the scope of this thesis, we established an experimental setup to study auditory spatial perception while recording the listeners brain activity using electroencephalography. We conducted two experiments on the encoding of sound source elevation in the human cortex. Both experiments results are compatible with the hypothesis that the cortex represents sound source elevation in a population rate code where the response amplitude decreases linearly with increasing elevation. Decoding of the recorded brain activity revealed that a distinct neural representation of differently elevated sound sources was predictive of behavioral performance. An exploratory analysis indicated an increase in the amplitude of oscillations in visual areas when the subject localized sounds during eccentric eye positions. More research in this direction could help shed light on the interactions between the visual and auditory systems regarding spatial perception. The experiments presented in this dissertation are, to our knowledge, the first studies that demonstrate the encoding of sound source elevation in the human cortex by using a direct measure of neural activity (i.e., electroencephalography).:Abstract . . . . . . . . . . . . . . . . . . . . . . 1 Zusammenfassung . . . . . . . . . . . . . . . . . . . . . . 7 1 Electroencephalography 13 1.1 Event Related Potentials and Oscillations . . . . . . . . . . . . 13 1.2 Comparison to other Methods . . . . . . . . . . . . . . . . . . . 14 1.3 EEG Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4.2 Referencing . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.4.3 Eye Blinks . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.4 Epoch Rejection . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5.1 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.5.2 Nonparametric Permutation Testing . . . . . . . . . . . 26 1.5.3 Source Separation . . . . . . . . . . . . . . . . . . . . . . 28 2 Sound Localization in the Brain . . . . . . . . . . . . . . . . . . . . 31 2.1 The Spatial Perception of Sound . . . . . . . . . . . . . . . . . . 32 2.1.1 Interaural Cues . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.2 Spectral Cues . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 Brain Mechanisms for Sound Localization . . . . . . . . . . . . 37 2.2.1 Auditory Pathway . . . . . . . . . . . . . . . . . . . . . 38 2.2.2 Extracting Localization Cues . . . . . . . . . . . . . . . 40 2.2.3 Neural Representation of Auditory Space . . . . . . . . 42 2.2.4 The Dual Pathway Model . . . . . . . . . . . . . . . . . 45 2.2.5 A Dominant Hemisphere for Sound Localization? . . . 47 2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3 A Free Field Setup for Psychoacoustics 51 3.1 Design of the Experimental Setup . . . . . . . . . . . . . . . . . 51 3.1.1 Loudspeakers . . . . . . . . . . . . . . . . . . . . . . . . 54 3.1.2 Processors . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.3 Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.4 Coordinate Systems . . . . . . . . . . . . . . . . . . . . 56 3.2 Operating the Setup . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.2.1 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.2.2 Loudspeaker Equalization . . . . . . . . . . . . . . . . . 59 3.3 Head Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . 61 3.3.1 Landmark Detection . . . . . . . . . . . . . . . . . . . . 62 3.3.2 Perspective-n-Point Problem . . . . . . . . . . . . . . . 62 3.3.3 Camera-to-World Conversion . . . . . . . . . . . . . . . 63 3.4 A Toolbox for Psychoacoustics . . . . . . . . . . . . . . . . . . 64 4 A Linear Population Rate Code for Elevation 67 4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 69 4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 70 4.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 70 4.2.3 Elevation Encoding . . . . . . . . . . . . . . . . . . . . . 72 4.2.4 Effect of Eye-Position . . . . . . . . . . . . . . . . . . . . 74 4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5 Decoding of Brain Responses Predicts Localization Accuracy . . . 81 5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 82 5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 83 5.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 84 5.2.3 Decoding Brain Activity . . . . . . . . . . . . . . . . . . 86 5.2.4 Topography of Elevation Encoding . . . . . . . . . . . . 88 5.2.5 Elevation Tuning . . . . . . . . . . . . . . . . . . . . . . 89 5.2.6 Hemispheric Lateralization . . . . . . . . . . . . . . . . 91 5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 A Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 B Publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11